Papers with natural language systems
A Computational Framework for Slang Generation (2021.tacl-1)
Copied to clipboard
| Challenge: | Existing language models trained on large text corpora are biased toward formal language and under-represent slang. |
| Approach: | They propose a framework that models the speaker’s word choice in slang context by relating the conventional and sexist senses of a word while incorporating syntactic and contextual knowledge. |
| Outcome: | The proposed framework outperforms state-of-the-art language models and better predicts the historical emergence of slang word usages from 1960s to 2000s. |
Toward Informal Language Processing: Knowledge of Slang in Large Language Models (2024.naacl-long)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have offered a strong potential for natural language systems to process informal language. |
| Approach: | They propose to use movie subtitles to evaluate slang in large language models . they find that smaller LLMs finetuned on the dataset achieve comparable performance . |
| Outcome: | The proposed dataset can be used to evaluate LLMs on slang detection and identification of regional and historical sources for interpretive insights. |
RetroGAN: A Cyclic Post-Specialization System for Improving Out-of-Knowledge and Rare Word Representations (2021.findings-acl)
Copied to clipboard
| Challenge: | Retrofitting is a technique used to move word vectors closer together or further apart in their space to reflect their relationships in a Knowledge Base (KB). |
| Approach: | They propose a system that uses two GANs to learn a one-to-one mapping between concepts and retrofitted counterparts. |
| Outcome: | The proposed system performs well on word-similarity benchmarks and a sentence simplification task. |
Unlearning Bias in Language Models by Partitioning Gradients (2023.findings-acl)
Copied to clipboard
| Challenge: | Recent research has shown that large-scale pretrained language models exhibit issues relating to racism, sexism, religion bias, and toxicity in general. |
| Approach: | They propose a gray-box method for debiasing pretrained masked language models using partitioned contrastive gradient unlearning (PCGU) aims to optimize only the weights that contribute most to a specific domain of bias by computing a first-order approximation based on the gradients of contrastive sentence pairs. |
| Outcome: | The proposed method is low-cost and can pinpoint the sources of social bias in large pretrained language models. |
Rarely a problem? Language models exhibit inverse scaling in their predictions following few-type quantifiers (2023.findings-acl)
Copied to clipboard
| Challenge: | Current work suggests that language models deal poorly with quantifiers-they struggle to predict which quantifier is used in a given context and also perform poorly at generating appropriate continuations following logical quantifier. |
| Approach: | They propose to use 960 English sentence stimuli to build 22 autoregressive transformer models of different sizes to test their performance on ‘few’-type quantifiers. |
| Outcome: | The proposed models perform poorly on ‘few’-type quantifiers, and the larger the model, the worse its performance. |